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(54) Dynamically relocatable tileable displays 

(57) A plurality of tileable devices for transferring 
data, comprising a first device having a display, a proc- 
essor, and a first communication module for transferring 
data, a second device having a display, a processor and 
a second communication module for transferring data, a 
third device having a display, a processor and a third 
communication module for transferring data, wherein 
the first device is connected in substantially simultane- 
ous communication with the second device and the third 
device to pass data based on spatial positions of the 
respective first device, second device, and third device. 
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Description 

[0001] Reliably, quickly, and intuitively transmitting 
complex commands to small portable computers can be 
difficult. Small computer devices do not generally have 5 
sufficient computer processing power to respond relia- 
bly to voice or handwritten (pen based) commands, key- 
boards are often absent or too small for accurate finger 
input, and conventional buttons are too large or support 
too limited a command instruction set User interface 10 
techniques that rely, on bulky external modules (full size 
infrared linked keyboards, tethered data gloves, or cam- 
era based gestural recognition equipment, for example) 
are expensive, often not readily available outside 
selected sites, and probably too awkward for wide- is 
spread use in conjunction with consumer level portable 
computing devices. 

[G002] User interface designers for portable comput- 
ers have attempted to compensate for some of these 
problems by constructing devices that rely on various 20 
spatial, positional, or environmental cues that manually 
or automatically allow for activation of various modes in 
the device. For example, some laptop computers use 
the action of opening/closing the lid to initiate automatic 
bootup/powerdown of the computer without requiring 25 
any additional signal input (e.g. such as depressing a 
"start" button or typing "I_o_g_o_n M on a keyboard) from 
a user. Alternatively, the use of small portable comput- 
ers that automatically switch control modes depending 
on position, orientation, or room location have been 30 
investigated. Buttonless manual control of a portable 
computer through deliberate user actions such as tilting 
the portable computer have also been described. 
[0003] However, all these solutions for interfacing with 
small portable computers have generally been limited in 35 
scope and functionality. What is needed is a user inter- 
face system suitable for even very small portable com- 
puters (having volumetric dimensions on the order of a 
one cubic centimeter) that is powerful, can be intuitively 
operated by an ordinary user with little training, and is 40 
still readily capable of modification or extension by the 
user. The present invention meets these requirements 
by providing a manipulatory user interface that 
responds to a user twisting, folding, bending, squeez- 
ing, shaking, tilting, spinning, lifting, or otherwise physi- 45 
cally manipulating the computer. 
[0004] In the manipulatory user interface system of 
the present invention, the most basic level of manipula- 
tion is known as a "senseme". A senseme is defined as 
a single indivisible type of physical manipulation. A par- so 
tial list of categories of sensemes include material 
transformations such as squeezing, twisting, stretching; 
local spatial transformations such as translation, rota- 
tion, orbiting; and environmental transformations based 
on temperature, light level, or vibration. For example, a 55 
small portable computer may support a deformable 
piece having multiple embedded sensors that detect 
folding, twisting, or bending of the deformable piece by 



a user. This computer can also contain a number of 
accelerometers that sense relative spatial information; 
gyroscopic, radio or infrared positional sensors for 
determining absolute position; and various thermal or 
photosensors that respectively detect temperature and 
light level changes. Intentional or unintentional modifi- 
cations detected by one or more of these sensor sys- 
tems can provide the basis for a powerful user interface 
scheme. 

[0005] As will be appreciated, each senseme category 
contains many individually distinguishable members. 
For example, consider the category of senseme known 
as a "pinch", a structural transformation generally com- 
pleted by a user squeezing the deformable piece 
between a forefinger and thumb. A pinch can be modi- 
fied by varying its speed (quick or slow pinch), magni- 
tude/intensity (light or hard pinch), portion of 
deformable piece pinched (top, bottom, or center of 
deformable piece pinched), or even portion of body 
used to pinch (right handed pinch or left handed pinch), 
with each modification being distinguishable as a 
senseme capable of being mapped onto a computer 
control command. 

[0006] Although the wide variety of easily distinguish- 
able sensemes would alone provide a powerful user 
interface to a computer, the present invention further 
extends the flexibility of the senseme based user inter- 
face by supporting computer control based on a "mor- 
pheme" input. The morpheme is a temporally 
synchronous (or overlapping asynchronous) tuple of 
one or more sensemes. Note that a morpheme can 
(and often will) contain more than one senseme. The 
sensemes combined into a morpheme can come either 
from the same category (the user pinches with a right 
hand while tapping with a left hand finger), or different 
categories (the user pinches the deformable piece with 
a right hand while modifying the spatial position of the 
portable computer by tilting it forward). 
[0007] Any morpheme can in turn be extended by par- 
ticipation in a "sentence". A sentence is defined as a 
sequence of one or more temporally disjoint mor- 
phemes. The sentence level allows definition of a phys- 
ical manipulatory grammar by appropriate choice of 
morpheme sequence, and corollary rules governing, for 
example, use of active (verb like) morphemes, naming 
(noun) morphemes, or connectors. Other possible 
grammar constructs used in sentences may include 
those based on "home" systems. Home systems are 
general-purpose gestural languages, whose grammar 
and syntax are not borrowed in any way from a host lan- 
guage. Examples of these languages are gestural lan- 
guages developed by deaf children of hearing parents 
who have not been exposed to American Sign Lan- 
guage (ASL), and the "plains talk" of North American 
Indians, which was used as a trade language. 
[0008] Accordingly, the present invention provides a 
method for inputting information to a computer con- 
nected to a deformable piece that can be manipulated, 
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and optionally to various position sensors (both relative 
and absolute), pressure sensors, thermal sensors, or 
even light sensors. The method comprises the steps of 
manipulating the deformable piece to provide a first 
morpheme input to the computer, with the first mor- 5 
pheme input normally triggering a first default action by 
the computer. The deformable piece may also be 
manipulated to provide a second morpheme input to the 
computer, with the second morpheme input converting 
the normally triggered first default action to a second w 
action. The first and second morphemes (and any sub- 
sequent morphemes) together form a sentence that can 
be interpreted as a command to implement a computer 
controlled action, whether it be to unlock an electroni- 
cally controlled door, display a graphical image on a 75 
computer display, or begin logging on to a computer net- 
work. Advantageously, such a user interface system is 
well suited for interaction with small computer devices, 
and may even be cross-cultural to a limited extent, with 
ability to squeeze or pinch being universal human 20 
attributes. 

[0009] The present invention is particularly useful for 
portable computers that can be held in the palm of a 
hand. If the portable computer is partially or completely 
surrounded with a deformable material having embed- 25 
ded or contact pressure/deformation sensors, a user is 
able to gesture with or manipulate the whole computer 
in order to achieve some desired result. Material defor- 
mation can be implemented at various scales. For 
instance, a computer and connected deformation sen- 30 
sors mounted inside a flexible tube could respond to 
right angle bending of the tube, or even complex knot- 
ting or looping of the tube. In practice however, only 
minute surface deformations are required, being just 
sufficient to provide tactile feedback to pressing, pinch- 35 
ing. or bending type manipulations. In either case, how- 
ever, the measurement of location and pressure applied 
to a surface is sufficient to characterize the mode of 
interaction (distinguishing a pinch from a prod). 
[0010] One particularly preferred embodiment of a *o 
handheld portable computer that responds to a physical 
manipulatory grammar in accordance with the present 
invention includes a computer, a feedback module to 
provide visual, auditory, or tactile feedback to a user 
(e.g. ( processor connected LCD display, audio speaker, 45 
or tactile display to present Braille or other conventional 
touch interface), and co-mounted graspable, deforma- 
ble piece partially or completely surrounding the feed- 
back module. In addition, various thermal or pressure 
sensors are mounted to detect handedness of a user, so 
with the grasping hand generally being the non-domi- 
nant hand for the user. Depending on the handedness 
of the user (which can be considered as a morpheme), 
the displayed data structure is modified. For example, 
text displayed on an LCD display screen may be auto- 55 
matically shifted rightward on the screen to allow pen 
based annotation on the left side of the screen, thereby 
aiding left handed users. 



[001 1 ] Physically manipulatable user interfaces addi- 
tionally provide an opportunity for multiple devices to 
interact in a user friendly manner. For example, a tilea- 
ble display system having multiple displays with touch 
sensitive deformable pieces surrounding the display 
can be used to organize documents based on relative 
position of contact of each display. For example, if two 
displays initially showing separate data structures (e.g 
two different pages from two different electronic books) 
are brought together in side by side touching contact, 
the displayed visual information can alter (e.g. by dis- 
playing adjacent pages of a single electronic book). As 
will be appreciated, one can also use multiple comput- 
ers connected to physically manipulatable control ele- 
ments to construct complex commands for organizing 
data structures. 

Figure 1 is a schematic diagram of an approxi- 
mately spherical hand holdable portable computer 
having a deformable surface, a status display, and a 
pressure sensor array for detecting surface defor- 
mations; 

Figure 2 is a graphical diagram illustrating various 
physical manipulation morphemes, with axes 
respectively illustrating complexity of senseme 
tuples required to form a morpheme, and plasticity 
of a device required to support the physical manip- 
ulation; 

Figures 3-16 schematically illustrate various pre- 
ferred classes of physical manipulation mor- 
phemes; 

Figure 17 is a graphical diagram illustrating various 
spatial morphemes, with a first axis illustrating com- 
plexity of senseme tuples required to form a mor- 
pheme, and a second axis illustrating the degree of 
positional information needed to support the physi- 
cal manipulation (moving along a continuum from 
relative local measurement along a single specified 
dimension to absolute global measurement with six 
degrees of freedom determined); 

Figures 18-26 schematically illustrate various pre- 
ferred spatial manipulation morphemes; 

Figure 27 is a graphical illustration showing increas- 
ing degrees of sensor system complexity that can 
be used to detect various categories of environ- 
mental stimuli, including light effects, thermal 
effects, the electromagnetic environment, and the 
vibratory/acoustic environment; 

Figure 28 is a graphical diagram illustrating various 
physical manipulation morphemes for multiple 
interacting devices, with axes respectively illustrat- 
ing the complexity of senseme tuples required to 
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form a morpheme, and increasing level of physical 
contact; 

Figures 29-34 schematically illustrate various pre- 
ferred manipulation morphemes for multiple inter- s 
acting devices; 

Figures 35 and 36 are schematic diagrams illustrat- 
ing "squeeze" and "tilt" morphemes applicable to a 
portable computer; ?o 

Figure 37 is a schematic diagram illustrating tilt and 
squeeze morphemes used to control view of large 
two dimensional data sets with a relatively small 
display of a portable computer; is 

Figure 38 is a schematic illustration representing a 
portable computer having a display ready to receive 
annotations from a right handed user; 

20 

Figure 39 is a schematic illustration representing a 
portable computer having a display ready to receive 
annotations from a left handed user; 

Figure 40 is an electronic schematic illustrating 25 
components of the pressure and tilt sensitive mod- 
ules of a portable computer such as illustrated in 
Figures 35-39. 

Figures 41 and 42 are schematic illustrations of a 30 
scanner/printer/copier using a paper shaped dis- 
play interface to support morphemic input; 

Figures 43-45 are schematic illustrations of tilable 
displays capable of supporting morphemic input; 35 

Figure 46 illustrates optical sensors and patterns 
suitable for use in conjunction with tileable displays 
such as illustrated in Figures 43-45; 

40 

Figure 47 illustrates radio transponders suitable for 
use in conjunction with tileable displays such as 
illustrated in Figures 43-45; and 

Figure 48 illustrates addressing of multiple tileable 45 
displays. 

[0012] Figure 1 illustrates an embodiment of the 
present invention suitable for supporting a morphemic 
user interface grammar. Support of the grammar can so 
require detection of a user's physical manipulation of a 
device, detection of relative or absolute spatial location 
of the device, detection of various environmental factors 
acting on the device, and even detection and interaction 
with multiple devices or external computer networks. As ss 
illustrated, a device 1 0 has a deformable surface 20 with 
an underlying deformation sensor mesh 22 for detecting 
surface deformation across or within multiple subre- 



gions of the deformable surface 20. The deformation 
sensor mesh 22 is connected to an internally contained 
processor 24 having associated memory system 26. 
For detecting various positional or environmental varia- 
bles, a sensing system 28 is also provided. The illus- 
trated device further includes a feedback module 33, 
which may include an externally visible status display 30 
or a non-visual feedback module 31 (typically delivering 
auditory or tactile feedback). In the illustrated device, a 
communications system 32 for reception or transmis- 
sion of information to other electronic or computing 
devices is also provided. All these components can be 
powered by a power supply 25, which is usually an inter- 
nally mounted rechargeable battery of conventional 
construction. 

[001 3] Although the device 1 0 is illustrated as having 
an approximately spheroidal and unitary mass, various 
other shapes are contemplated to be within the scope of 
the present invention. For example, the overall shape 
may be similar to various rectangular prisms, or can be 
ellipsoidal, toroidal, planar, or even be malleable 
enough to support a wide range of user defined irregu- 
lar shapes. In addition, multiple cooperating shape ele- 
ments are contemplated using conventional designs 
that permit interlocking of multiple shape elements (e.g 
using a ball and socket, a lock and key, or slidabie or 
rotatable interlocked components). 
[001 4] Whatever the shape of device 1 0, for operation 
of the present invention the device 10 is completely or 
partially enveloped by the deformable surface 20. The 
present invention supports use of a great variety of 
designs and materials for the deformable surface 20, 
depending on the required plasticity, durability, longev- 
ity, and of course, cost constraints. For example, con- 
templated designs for deformable surface 20 include, 
but are not limited to: 

a closed or open celled polymeric foam material 
having a wall thickness of millimeters to centime- 
ters, with thinner walled embodiments being sup- 
ported (e.g. by adhesive attachment) by an internal 
hard shell (constructed from polymeric or metallic 
materials), and those thicker walled embodiments 
directly supporting (by, e,g. brackets or supports) 
internal components such as processor 24. Suita- 
ble foams may include those composed in whole or 
in part of widely available synthetic rubbers such as 
polychioroprene (neoprene), polystyrenes, rubber 
or nitrile rubber latex foams, polysiloxanes, block 
polymers including styrene-butadiene or styrene 
isoprene, or any other conventional materia! having 
good elasticity and deformability; 
a thin single layer polymeric surface loosely 
wrapped around a internal hard shell (the hard shell 
being constructed from polymeric or metallic mate- 
rials). For example, a nylon or cotton weave, single 
layer polyethylene, synthetic rubber (with little or no 
foam cells present), or natural polymeric materials 
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such as leather wrapped around a polystyrene cas- 
ing can be used; 

a composite layered surface having a durable poly- 
meric outer layer supported by an inner foam layer; 
or even 5 
a polymeric bilayer having an intermediate fluid or 
gel layer of a viscous or thixotropic material that can 
be used to support extreme deformations. The 
intermediate layer can be relatively thick (one the 
order of centimeters), or in certain embodiments 10 
can have a thickness measured on micron to millim- 
eter scales. Such extremely thin layers would allow 
complex twisting, folding, curling, or crumpling 
actions, and have been described in conjunction 
with U.S. Patent 5,389,945, assigned to Xerox 15 
Corp., the disclosure of which is herein specifically 
incorporated by reference. 

[0015] The deformation sensor mesh 22 can be 
embedded within, or positioned to contact, the deforma- 20 
ble surface 20. The deformation sensor mesh 22 can 
include an array of individual compressional or ten- 
sional strain sensors, or alternatively, embedded or 
attached positional sensors. For certain applications, 
continuous sensors (e.g. bilayer sheets of capacitance 25 
sensors) may be employed. One particularly useful con- 
tinuous sensor type uses multiple capacitance or resist- 
ance strips, with deformation pressure resulting in a 
positionalfy localizable analog signal proportional to the 
applied deformation pressure. Various sensor types can 30 
be used, including simple capacitance sensors, resis- 
tive strain sensors, analog or digital pressure switches, 
inductive sensors, or even fluid flow sensors. Depend- 
ing on the sensor type employed, sensor data can be 
directly fed to the processor 24 in digital form, or be 35 
transformed to digital format by an general purpose 
analog/digital converter that typically provides a 4 or 8 
bit range (although as few as one or as many as 32 bits 
may be required by various applications). An analog to 
digital converter may be internal to the processor 24 or 40 
provided as an external module. As will be appreciated, 
the sensor mesh 22 is intended to include combinations 
of sensors and sensor types, which can be used over 
the whole or part of the deformable surface 20. 
[0016] A positional or environmental sensor system 45 
28 can also be supported by device 10. Various sensor 
modes can be supported, including absolute or relative 
positional information as determined by gyroscopic sen- 
sors, accelerometers, or acoustic or infrared ranging 
techniques. Environmental sensors, including conven- so 
ttonal light, image, thermal, electromagnetic, vibratory, 
or acoustic sensors can also be present. Depending on 
the desired application, even costly environmental or 
positional sensors such as those incorporating differen- 
tial GPS positioning, image analysis or recognition, 55 
acoustic or voice identification, or differential thermal 
sensors can be used as morphemic input. Such mor- 
phemic input, taken in conjunction with morphemic input 



as detected by sensor mesh 22. can enhance precision 
and flexibility of a user's control of device 10. 
[0017] As illustrated, both sensor system 28 and sen- 
sor mesh 22 are connected to the processor 24 and 
associated memory 26. The processor 24 and memory 
26 are typically mounted within the deformable surface 
20, by either direct attachment to the deformable sur- 
face 20 or by attachment to a hard casing positioned 
within the deformable surface 20. Conventional CISC or 
RISC processors can be used in the illustrated embodi- 
ment, with low power processors such as the Signetics 
87c752 or 87c751, Motorola 68HC1 1 or 68582, or ARM 
710 being preferred, if convenient, coprocessors such 
as analog to digital converters or digital signal proces- 
sors can be used alone or in conjunction with a main 
processor. Conventional flash, static, or dynamic RAM 
can used in the present invention, although for certain 
applications higher cost embedded DRAM may also be 
used. In some storage intensive applications, memory 
26 can include additional harddisk storage, either 
located within the device 10 or available through an 
external connection. As will be appreciated, for many 
applications use of optional external communications 
can at least partially supplant use of internal processors 
and memory (except for that necessary to support 
required sensor or communication buffering and signal- 
ling). 

[001 8] The present invention optionally supports com- 
munications with an external computer system 40 using 
its internal communications system 32 and associated 
transceiver 34, The external computer system 40 also 
includes a transceiver 42, a personal computer or work- 
station 44, and is connected to a local or wide area net- 
work computer system 46. The transceivers 34 and 42 
can support various communication protocols and 
designs, including use of a serial tethered line 36 (using, 
for example the RS-232C interface protocols), use of 
infrared signals 38 adhering to widely utilized IRDA 
communication standards, or use of radiofrequency sig- 
nal 37 (which can be, for example, a cellular telephone, 
900MHz radio, or digital PCS telephonic communica- 
tions). Alternative communication standards, or even 
alternative communication carriers such as those based 
on optical or acoustic techniques, can of course be 
employed. 

[0019] As will be appreciated, in addition to direct 
communication with external computer system 40. the 
device 10 can be directly or indirectly maintained in con- 
tinuous or intermittent communication with a number of 
suitably equipped electronic devices, including a tablet 
computer 110. or even a physically manipulatable port- 
able computer 11 similar in design and function to 
device 10. Communication can be direct to a target 
device, or through an intermediary retransmitter such as 
computer system 40. Other possible communication 
targets include automation control systems, security 
authorization units, personal digital assistants, note- 
book computers, or any other suitably equipped elec- 
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tronic system. 

[0020] Results of communications with external 
devices, presentation of device stored information, or 
device status updates can all be provided to a user 
through processor 24 controlled update of feedback s 
module 33. Feedback to a user can be primarily visual, 
such as can occur in conjunction with visual display 30. 
Generally, the display 30 can be a conventional passive 
or active matrix liquid crystal display, although use of 
more sophisticated (and expensive) displays based on 10 
various electroopticai or micromechanical techniques 
can of course be used. In addition, for certain devices a 
non-imaging display such as may be formed by a small 
number of status lights (e.g. red or green LEDs), or 
localized or distributed chromatic changes (in conjunc- 15 
tion with a deformable surface 22 constructed with suit- 
able electrochromic materials) may be all that is 
necessary for visual feedback to the user. 
[0021] In some embodiments of the invention, visual 
output through display 30 may be augmented (or even 20 
replaced) with a non-visual display 31. The non-visual 
display 31 can include tactile displays based on internal 
actuators, auditory feedback, or even displays based on 
conformal changes in device appearance. For example, 
one possible feedback display is based on internal audi- 25 
tory speakers (emitting a range of sounds from simple 
"beeps" to well formed speech, depending on available 
processor speed and functionality) for providing user 
feedback. As will be appreciated, non-visual display 31 
and its associated actuators or electronics can support 30 
alternative feedback modes, including, for example, 
force feedback to a user through internal actuators, tac- 
tile based feedback (e.g with multiple surface projec- 
tions for presentation of Braille or other conventional 
tactile user interface), modifications to the surface tex- 35 
ture of the device, or any other conventional mechanism 
for supplying information to a user. 
[0022] To better appreciate operation of the present 
invention, some selected modes of physical manipula- 
tion of device 10 are schematically presented in Figure *o 
1 . As seen in Figure 1, device 10 can be translationally 
moved in three spatial dimensions, as illustrated with 
reference to orthogonal force arrows 50, 51, and 52. In 
addition to translational motion, the device 10 can be 
rotationally moved in any or ail three spatial dimensions, 45 
as represented by arrows 53, 54, and 55. By use of sen- 
sor system 28 (alone or in combination with communi- 
cation system 32), the relative or absolute position and 
orientation in three dimensions can be determined. 
[0023] In addition to determining spatial position and so 
orientation through use of sensors 28, the device 1 0 can 
optionally use sensor mesh 22 to measure and localize 
transient or continuing force application, with force vec- 
tors and related timing information being determined 
and interpreted. A few possible force actions (deforma- ss 
tion modes) are schematically illustrated in Figure 1, 
with arrows 60 and 61 illustrating depression of surface 
20 (with the combination representing a squeeze), 



arrows 62 and 63 illustrating sliding or rubbing deforma- 
tion (with the combination representing a twist), and the 
combination of sliding arrows 65 and 66, and outward 
pull 67, together representing a pinch and outward pull. 
The strength of the applied force can be measured (e.g. 
a hard or soft squeeze is differentiated), its spatial 
expanse found (e.g. to differentiate between poking with 
a fingertip or a thumb pad), and timing determined (e.g. 
to differentiate between a quick or a slow depression of 
the surface). The deformation so caused may be either 
permanent or transitory. 

[0024] As will be appreciated by those skilled in the 
art, each of the foregoing force actions represented by 
arrows can be considered a senseme. Some temporally 
distinguishable sensemes (or combinations of sense- 
mes such as the foregoing discussed pinch/pull combi- 
nation) further represent morphemes used as a basis 
for a morphemic grammar in accordance with the 
present invention. All of the following described mor- 
phemes can be modified by numerous variations in 
applied pressure, force utilized, appendages used, body 
parts, or extraneous intermediary objects used to apply 
force. In addition, timing of various objects (whether 
quick, slow, or alternately quick and slow) can modify 
interpretation of a morpheme. For example, if "squeeze" ' 
is taken as a typical morpheme, one can appreciate var- 
ious squeeze operations such as quick squeeze, slow 
squeeze, hard squeeze, soft squeeze, narrow squeeze, 
wide squeeze, squeeze between hands, squeeze 
between one hand and a user's chest or head, squeeze 
between one hand and a table or wall, a squeeze made 
between two pens or two books, or even a squeeze 
between a user's tongue and the roof of the mouth. For 
purposes of the present invention, ail such squeeze 
morphemes would be considered members of the 
"squeeze" class, with individual variations acting as 
possible modifiers or selected cases, just as a "house 
cat" might be considered a particular member of the 
class of "felines", which more generally embraces lions, 
tigers, and bobcats. 

[0025] To aid in understanding the diversity of contem- 
plated physical manipulation morphemes, Figure 2 illus- 
trates selected morphemes arranged by increasing 
plasticity of the device required to enable morpheme uti- 
lization, and by increasing complexity of available 
senseme tuples required to form or interpret a mor- 
pheme applied to a particular class of device. Beginning 
with the least plastic device and the simplest senseme 
set used to compose a morpheme, a definition of a pos- 
sible physical manipulation and typical function invoked 
by that manipulation of a device similar (but of course 
possibly more complex) to that described in conjunction 
with Figure 1 is presented: 

DEPRESS 

[0026] Definition: Indenting one or more subregions of 
the device through the application of pressure. 
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[0027] Example: As seen in Figure 3. consider a 
device 122 having a display 123. The device 122 sup- 
ports a graphical illustration software application which 
allows users to position geometric shapes or user 
defined graphical objects. The device can have four 
pads 124 placed around it, one on each side. By 
deforming a particular side, the user indicates a desire 
to "nudge" the currently selected geometric shape 125 
away from that side to a new position 126. 

SQUEEZE 

[0028] Definition: To deform one or more subregions 
of a device through the application of vectors of force, 
some components of which are directed towards each 
other, which compress the morphological structure of 
the device. This is a special case of depression. 
[0029] Example: As illustrated in Figure 4, consider a 
device 132 which can display one or more documents in 
either iconic (closed) or textual display (open) modes. 
By selecting an open document 135, and then squeez- 
ing a deformable edge 134 of the device 132, the user 
indicates a desire to make the document "smaller, in 
this case to iconize it as icon 136. 

FOLD 

[0030] Definition: To deform a second subregion by 
bending the first subregion such that it partially or com- 
pletely overlaps the second subregion. Further defor- 
mations can be applied to that new morphological 
structure on other subregions. 
[0031] Example: As illustrated in Figure 5, consider a 
device 142 which can display documents. Suppose this 
device 142 has been augmented such that a deforma- 
ble horizontal "flap" 1 44 on a top edge of the device 142 
can be folded to partially obscure a display 143. When 
the user makes this folding gesture, the user indicates a 
desire to password-protect ("hide") the currently dis- 
played document 

CURL 

[0032] Definition: Deforming one or more subregions 
of the device by spirally configuring subregions relative 
to each other in a cylindrical or circular morphological 
structure. 

[0033] Example: As illustrated in Figure 6. consider a 
device 1 50 which can display documents in various lan- 
guages (English, French, etc.). When the user takes 
such a device 150, curls in a direction indicated by 
arrow 157 into a tube, and then uncurls it, this "abra- 
cadabra" gesture tells the device 150 to display the cur- 
rent document in a different language. 

STRETCH 

[0034] Definition: Deforming one or more subregions 



of a device through the application of vectors of forces, 
some components of which are directed away from 
each other, the vectors being applied at opposite ends 
of the device. 

5 [0035] Example: As illustrated in Figure 7, consider a 
device 160 with a graphical software application which 
allows users to manipulate geometric shapes. By 
stretching the device 160, the user indicates a desire to 
"resize" or "rescale" the currently displayed shapel 65 to 

io a larger size 166, the amount of resizing being a func- 
tion of the amount of deformation. Note that SQUEEZ- 
ING can indicate resizing the currently displayed to a 
smaller size. 

15 PINCH 

[0036] Definition: Manipulating one or more subre- 
gions by applying vectors of forces, aligned directly 
towards each other, on opposite sides of the affected 
20 subregion(s). This is typically, but not exclusively, 
accomplished using two-finger tactile force. PINCH is a 
special case of SQUEEZE. 

[0037] Example: As illustrated in Figure 8, consider a 
device 170 which can copy documents. By performing a 
25 "pinching" action 1 75, the user indicates that they wish 
the next set of copies to be issued in stapled form. 

DOGEAR 

30 [0038] Definition: Deforming a second subregion by 
folding a first subregion at a logical corner or edge of the 
second subregion, indicating a marker or location point 
(e.g. a bookmark) to be later referenced. DOGEAR is a 
special case of FOLD. 

35 [0039] Example: As illustrated in Figure 9, consider a 
device 1 80 which displays a subset of the pages from a 
multi-page document. By "dogearing" the upper right 
corner 185 of device 180, the user indicates that they 
wish a bookmark associated with the currently dis- 

40 played page or pages. 

TWIST 

[0040] Definition: Deforming one or more subregions 
45 of the device through the application of two opposing 
rotational forces offset from each other by a non-zero 
difference about some central axis. 
[0041] Example: As illustrated in Figure 10, consider 
a device 190 which, over time, becomes degraded in 
50 some aspect of its performance (its disk becomes frag- 
mented, its memory needs garbage-collection, etc.). By 
performing a "TWIST gesture 195, the user indicates 
that they wish the device to "wring itself out", perform- 
ing, for example, garbage collection. 

55 

RELIEF-MAP 

[0042] Definition: Deforming one or more subregions 
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of the device by raising and or lowering them by either 
spatial transformation or the addition/removal of mate- 
rial. 

[0043] Example: As illustrated in Figure 1 1 , consider 
a device 200 which can display documents in either s 
one- or two- page format. When the user "scores" the 
device by making a vertical indentation about the center 
axis of the device while it displays a single page 206, the 
device 200 interprets the morpheme to request display 
of documents in a two-page format as pages 207 and 10 
208. 

RIP 

[0044] Definition: Deforming one or more subregions is 
of the device by introducing a spatial discontinuity, by 
applying vectors of force to partially or totally disconnect 
these subregions from the device. 
[0045] Example: As illustrated in Figure 12, consider 
a device 210 which can copy some or all of its informa- 20 
tion. When the user performs the "ripping" gesture 215, 
removing one or more subregions, the device 210 cop- 
ies its currently selected data set onto those subregions 
216 and 217. 

25 

PERFORATE 

[0046] Definition: Deforming one or more subregions 
of the device by means of introducing a change in the 
spatial connectivity of the subregions such that a hole is 30 
introduced (either temporarily or permanently) in the 
device. 

[0047] Example: As illustrated in Figure 13, consider 
a device 220 which is used to route messages between 
various parts of a device network, and which displays 35 
this functionality to the user by means of lines 222 rep- 
resenting message pathways. When the user perforates 
the device 220 with a finger or object 224 to introduce a 
hole in one of these pathways 222, the system stops 
routing messages along that pathway. *o 

SIMILARITY 

[0048] Definition: Deformation of one or more subre- 
gions of a device previously configured to represent 45 
some other predefined object. Typically the device acts 
in a manner consistent with the behavior of the real- 
world object when it is manipulated in this manner. 
[0049] Example: As illustrated in Figure 14, consider 
a device 230 which contains text-to-speech and audio so 
input capability, and which presents itself to the user in 
an anatomically accurate shape of a human head. 
When the user opens lips 232 on the human head, inter- 
nal sensors detect opening of the lips and activate text- 
to-speech capability. 55 



3D MAP 

[0050] Definition: The morphing of a device that can 
be molded around an external object to permit a sensor 
mesh to determine simultaneously size and shape of 
the external object. The range of possible external 
objects is large but is limited by the size of the device's 
solid inner-housing and the volume of the moldable 
material in the outer housing. In this system the device 
has the ability to accurately sense the amount of mate- 
rial from its inner surface to the outer edge of the mold- 
able material (example: via ultrasonic sounding, similar 
to sonar in water) thus determining an accurate elec- 
tronic model for the shape of the molded enclosure. 
[0051] Example: As illustrated in Figure 15, by press- 
ing a device attached moldable material 244 around the 
surface of an external object (e.g. a cog 242) a device 
240 can automatically generate a CAD model of that 
object and store it in its memory. 

MIMICRY 

[0052] Definition: Deforming one or more subregions 
of the device such that the resultant morphological 
structure resembles a known real-world object and- 
through this association of subregions. the device acts 
in a manner consistent with the object it resembles. 
[0053] Example: As illustrated in Figure 16, consider 
a device 250 containing a computer which contains text- 
to-speech and audio input capability, and which 
presents itself to the user as a misshapen blob having 
the consistency and plasticity of moldable putty or clay. 
When the user performs the "MIMICRY" action by mold- 
ing part of the device 250 to resemble an ear, the audio 
input capability is activated. 

[0054] In addition to morphemes based on physical 
manipulation, various morphemes based on varying 
degrees of relative or absolute spatial positioning are 
contemplated to be useful in practice of the present 
invention. To aid in understanding the diversity of con- 
templated spatial morphemes, Figure 17 illustrates 
selected spatial morphemes arranged by increasing 
knowledge of spatial position required to enable mor- 
pheme utilization, and by increasing complexity of avail- 
able senseme tuples required to form or interpret a 
morpheme applied to a particular class of device. 
Beginning with a device having only rudimentary rela- 
tive positioning functionality for supporting simple spa- 
tial sensemes, and ending with a device absolutely 
positionable to within centimeters anywhere on Earth, a 
definition of a possible spatial manipulation and typical 
function invoked by that manipulation of a device similar 
(but of course possibly more complex) to that described 
in conjunction with Figure 1 is presented: 

TRANSLATE (relative to device) 

[0055] Definition: The linear movement of a device's 
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center of mass from one position in space to another. 
[0056] Example: Used to substitute for mouse control- 
led graphical "sliders' in conventional graphical user 
interfaces. "When it is only physically possible to display 
a small amount of list, large listings can still be searched 
by "scrolling" a display window in response to the 
TRANSLATE morpheme. 

SHAKE 

[0057] Definition: Spatially translating all subregions 
of a device by repeated movement in opposing direc- 
tions, such that the net translation is negligible. 
[0058] Example: As illustrated in Figure 18, consider 
a device 260 which is used as a calculating device. 
When the user performs the "SHAKE" gesture, the 
device 260 clears its accumulator. 

REVOLVE 

[0059] Definition: Rotating ail subregions of a device 
by rotating the subregions about a point internal to the 
device, about any arbitrary plane. 
[0060] Example: As illustrated in Figure 19, consider 
a device 270 which displays an imaged slice of volumet- 
ric data, such as medical data from a series of CAT 
scans. By rotating the device about a center point 272 
internal to the device 270 to a new position 274, the 
plane specifying the imaged slice is changed accord- 
ingly. 

TILT 

[0061] Definition: Rotating one or more subregions of 
a device by rotating the subregions such that one or 
more components of the rotary force are in the direction 
of gravity and the amount of rotation is between about - 
180 degrees and +180 degrees. 
[0062] Example: Consider a device which displays 
frames from an animation sequence on the side facing 
the user. As the device is tilted away, the speed of the 
animation increases - as its tilted towards the user, the 
speed of the animation decreases, analogous to opera- 
tion of a gas pedal. 

FLICK 

[0063] Definition: A forwards TILT immediately fol- 
lowed by an opposing backwards TILT. 
[0064] Example: As illustrated in Figure 20, consider 
a device 280 which can transmit some subset of its data 
to another device. When the user performs the "FLICK" 
gesture by quickly tilting in the direction of arrow 282, 
followed by a reverse tilt along arrow 282, the device 
280 performs this transmission, towards the device (not 
shown) pointed to by the ray of the gesture. 



SPIN 

[0065] Definition: Rotating one or more subregions of 
the device by rotating the subregions about a point inter- 
5 nal to the device, such that the plane of rotation is one 
of the device's surface planes. SPIN can be considered 
a special case of REVOLVE. 

[0066] Example: As illustrated in Figure 21 , consider 
a device 290 which can display a frame of video 295 

w from a video sequence. The user performs the "SPIN" 
gesture in a counter-clockwise direction, the device dis- 
plays earlier frames in the sequence: when the gesture 
is performed in a clockwise direction (arrow 292). the 
device 290 displays a later frame 296 in the sequence 

75 (represented by film strip 294). 

ORIENT 

[0067] Definition: Rotating one or more subregions of 
20 the device by rotating the subregions about the center of 
the device, such that the plane of rotation is one of the 
device's surface planes, and the amount of the rotation 
is a multiple of 90 degrees (i.e. to rotate the device 
between cardinal compass points). Orient can be con- 
25 sidered a special case of SPIN, which in turn is a spe- 
cial case of REVOLVE. 

[0068] Example: As illustrated in Figure 22 t consider 
a device 300 which can display a document in either 1 
page, 2-page, or 4-page format ("l-up", "2-up". or H 4- 

30 up"). When the user performs the orient gesture in a 
clockwise direction (arrow 302). the device 300 
increases the number of document pages it is display- 
ing from one page 305 to two pages 306 and 307. Fur- 
ther orient gestures would increase the number of 

35 displayed pages. When performed in a counter-clock- 
wise direction, the device 300 decreases the number of 
pages it is displaying. 

FACE 

40 

[0069] Definition: Manipulating one or more subre- 
gions of the device such that a first set of subregions is 
no longer bottom-most and a second distinct set of sub- 
regions now assumes the first subregions former posi- 
45 tion. 

[0070] Example: As illustrated in Figure 23. consider 
a device 310 which displays documents, and which 
allows users to edit such documents. Consider further 
the case in which the device presents itself to the user 

so in the form of a cube, in which 6 different documents are 
displayed on the 6 different faces. When the user per- 
forms the "FACE" gesture by making a particular face 
top-most, the document which is now on the top-most 
face becomes editable by the user, while the document 

55 which is no longer top-most is no longer editable. 
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LIFT 

[0071] Definition: The movement of a device's center 
of mass in a direction opposite to the current gravitation 
force acting on the device. 

[0072] Example: Commanding the device to display 
the computer's file system at a position one level higher 
in the hierarchy. 

PAN 

[0073] Definition: The application of a TRANSLATION 
to a device such that it is moved parallel to the front of 
the user's body, at a substantially constant height. 
[0074] Example: Viewing a spread-sheet on a device 
with a display so small only one cell can be shown. By 
PANing the device, the contents of the current row can 
be shown in sequence depending on the rate or amount 
of PAN. However, if the device were rotated away from 
its current orientation during the PAN, a new row would 
be chosen. The selection of the row could be dependent 
on the deviation from the original orientation. 

PUSH-PULL 

[0075] Definition: Manipulating one or more subre- 
gions of the device by spatially translating them such 
that they are moved along a line of projection from the 
center of the device to the vertical axis of the user's 
body. 

[0076] Example: Consider a device with audio output 
capability. As the device is "pushed" further away from 
the body, its audio output level increases. When it's 
"pulled" towards the body, its level decreases. 

WHACK 

[0077] Definition: The application of an accelerative or 
de-accelerative force to one or more subregions of a 
device such that said subregions contact or are con- 
tacted by an external object causing an equal and 
opposite countering force. 

[0078] Example: As illustrated in Figure 24, consider 
a device 320 which can perform long and unpredictable 
database searches. When the user performs the 
WHACK gesture (e.g. upon a table 322), the current 
search is aborted. 

ORIENT (relative to environment) 

[0079] Definition: Manipulating two subregions of the 
device such that the line drawn between the centers of 
those two subregions alters its orientation with respect 
to the surrounding environment. 
[0080] Example: Displaying a CAD drawing of a 
machine-part in 3D on the display of a mobile device. As 
the orientation of the device changes, so too does the 
viewing angle and position of the rendered image. 



ORBIT 

[0081] Definition: Rotating one or more subregions by 
rotating the subregions and/or the center of mass of the 

s device about some point exterior to the physical bound- 
aries of the device, about any arbitrary axis. 
[0082] Example: As illustrated in Figure 25, consider 
a device 330 which can search a network database, 
such as the World Wide Web, for information. When the 

10 user performs the "orbit" gesture, such a search is initi- 
ated. The radius 332 of the rotation 334 specifies the 
breadth of the search - wider circles specify a wider 
search. The speed of the gesture specifies the time limit 
imposed on the search - the quicker the gesture, the 

15 more cursory the search. 

ORBIT RELATIVE TO USER 

[0083] Definition: Rotating one or more subregions by 
20 rotating the subregions and/or the center of mass of the 
device about some point exterior to the physical bound- 
aries of the device, where said point is proximal to a 
body feature of the user. This is a special case of 
ORBIT. 

25 [0084] Example: As illustrated in Figure 26, consider - 
a device 340 which can perform audio output. By per- 
forming the ORBIT gesture (in direction indicated by 
arrow 344) about the user's ear 345, the audio output is 
activated. 

30 

MOVEMENT IN ROOM 

[0085] Definition: The local detection of a device's 3D 
position relative to reference points found within an 
35 enclosing room. Differences in the measured position 
are used to trigger actions. 

[0086] Example: A virtual filing system that allows you 
to save and restore files based on the devices current 
position in the room. To save a file you might think care- 

40 fully about the contents of the file and then walk to the 
position in the room that might be most easily associ- 
ated with it. When retrieving the file you would use the 
same thought processes and go back to the position 
you had associated with the file. On doing so the files 

45 associated with that position would be displayed and 
you would then be able to select the file you were look- 
ing for. The system is useful because the human mind is 
very good at remembering information that is spatially 
organized rather than in some abstract information data 

so structure. 

MOVEMENT BETWEEN WIDELY SEPARATED SITES 

[0087] Definition: Manipulating one or subregions of 
5$ the device such that the detected absolute spatial posi- 
tion of those subregions is changed. 
[0088] Example: Consider a device which can display 
information from a database of client information. When 
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the device is moved to a different client site, the device 
automatically updates its display to display information 
for the nearest client site. 

[0089] in addition to morphemes based on physical 
manipulation or spatial positioning, various morphemes 
based on sensed environmental conditions are contem- 
plated to be useful in practice of the present invention. 
To aid in understanding the diversity of contemplated 
environmental morphemes. Figure 27 illustrates 
selected environmental morphemes loosely arranged in 
order of increasing sensor complexity needed in some 
commonly sensed environmental categories. For each 
of the proffered categories, some selected sensing sys- 
tems supportable by a device such as that described in 
conjunction with Figure 1 are presented: 

LIGHT 

[0090] Definition: Manipulating one or more subre- 
gions of the device such that the amount of light falling 
upon those subregions changes. 
[0091] Example: Consider a device used in a lecture 
hall to take notes. When the room lights are turned on, 
the light sensors detect this and turn down the backlight 
to conserve energy. When the room lights are turned off 
(e.g. during a slide show) the light sensors detect this 
and turn up the backlight to increase viewability. 
[0092] Light sensors can range from thresholded 
binary light detectors, to light pattern detectors, to full 
imaging systems. Advanced techniques can include 
image analysis and recognition to identify objects or 
persons. 

HEAT 

[0093] Definition: Manipulating one or more subre- 
gions of the device such that the amount of heat applied 
to those subregions changes. 
[0094] Example: Consider a portable computer which 
has a stylus for entering text. By looking at the heat pro- 
file along the back surface of the computer, the compu- 
ter can detect whether it is being held with the left hand, 
the right hand, both hands, or neither hand, and update 
its interface accordingly. 

[0095] Thermal (heat) sensors can range from simple 
temperature sensors to sophisticated differential ther- 
mal mappers and thermal imagers. 

ELECTROMAGNETIC 

[0096] Definition: Manipulating one or more subre- 
gions of the device such that the electromagnetic spec- 
trum applied to those subregions changes. 
[0097] Example: By analyzing the radio spectrum, the 
device can derive estimates as to its absolute spatial 
position, and use that to alter its functionality. 
[0098] Electromagnetic detection can include mag- 
netic compasses, radio detection, or GPS signal detec- 



tion. More advanced techniques can include 
electromagnetic spectrum analysis and interpretation, 
such as roughly determining location based on availa- 
ble radio signals. 

5 

VIBRATE 

[0099] Definition: Manipulating one or more subre- 
gions of the device by vibration. 

w [0100] Example: Consider a device which displays 
textual information. When the user takes the device on 
the bus. the ambient vibration level sensed by the 
device changes, and the device increases the size of 
the displayed text to help the user compensate. 

15 [0101] This class of environmental morphemes can 
include detection of intermittent contacts, low frequency 
rumblings, or acoustic level detection. More advanced 
techniques requiring greater processor power include 
maximum frequency identification, spectral analysis of 

20 acoustic frequencies (enabling the device to distinguish 
background environmental noises from speech, for 
example), or even speech based identification of per- 
sons in the vicinity of a device. 
[0102] In addition to morphemes based on physical 

25 manipulation, spatial position, or sensed environmental 
factors, various morphemes based on cooperation 
between multiple interacting devices are contemplated 
to be useful in practice of the present invention. To aid in 
understanding the diversity of contemplated spatial 

30 morphemes. Figure 28 illustrates multidevice mor- 
phemes arranged by an increasing level of possible 
physical contact, and by increasing complexity of avail- 
able senseme tuples required to form or interpret a mor- 
pheme applied to a particular class of device. Beginning 

35 with a device having only rudimentary edge deformation 
functionality for supporting simple spatial sensemes, 
and ending with complex deformable or embeddable 
devices that can be wrapped about each other, a defini- 
tion of a possible multidevice manipulation and typical 

40 function invoked by that multidevice manipulation of 
devices similar (but of course possibly more complex) to 
thatdescribed in conjunction with Figure 1 is presented: 

TOUCH 

45 

[01 03] Definition: To move one or more subregions of 
a device such that they enter physical contact with a 
subregion of a second device, in any alignment and to 
any extent Or, to take two devices so aligned and 

so remove that alignment. 

[0104] Example: Consider two portable computers 
350 and 351 , the first of which contains a database, and 
the second of which contains an IRDA port. When the 
user touches the first computer 350 to the second com- 

55 puter 351, the database is transmitted via the second 
computer's port. 
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MATCH 

[01 05J Definition : To move one or more subregions of 
a device such that they enter physical contact with one 
or more subregions of a second device, with the subre- 
gion(s) of the first device and the subregion(s) of the 
second device being aligned along one or more edges. 
Or, to take two devices so aligned and remove that 
alignment. 

[0106] Example; As illustrated in Figure 30, consider, 
multiple devices 360, 361 , and 362, which contain differ- 
ent versions of the same basic database. When the 
user MATCHes the first device 360 to the second device 
361, followed by matching the third device 362 to the 
matched first and second devices, their databases are 
reconciled (synchronized). 

STACK . 

[0107] Definition: To move one or more subregions of 
a device such that they enter physical contact with a 
subregion of a second device, such that the first device 
is now located above, but physically adjacent to, the 
second device. Or, to take two devices so aligned and 
remove (i.e. unstack) that alignment 
[0108] Example: As illustrated in Figure 31 . consider 
a set of devices 370, 371 , and 372, each of which is dis- 
playing a frame of video from a longer video sequence. 
When the devices are stacked, the ordering of the 
stacking specifies an order for the video editing, and a 
single composite video is now produced. 

TILE 

[0109] Definition: To move one or more subregions of 
a device such that they physically contact with a subre- 
gion of a second device, such that the first device and 
second device now form a single seamless spatial unit 
or to take two devices so aligned and remove that align- 
ment. TILE is a special case of MATCH. 
[01 1 0] Example: As illustrated in Figure 32. consider 
a set of devices 380, 381, 382, 383, each of which can 
independently display a portion of a large photograph. 
When the devices are tiled, each device displays the 
portion of the photograph appropriate to its current rela- 
tive position in the tiled grid. 

RELATIVELY ALIGN 

[011 1 ] Definition: To move one or more subregions of 
a device such that they engage in a particular spatial 
relation to one or more other devices, where said 
devices are not touching. 

[011 2] Example: As illustrated in Figure 33, consider 
a set of devices 390, 391, 392, and 393 which are dis- 
playing a multi-page document Whichever device is 
presently placed at the far left (device 390) displays the 
table of contents, whichever one is presently placed at 



the far right (device 393) displays the index, and the oth- 
ers display pages according to their respective loca- 
tions. As different devices can have different display 
capabilities, moving them about can alter the document 

5 display. For example, if only one of the devices has a 
color display, when it is moved from second position to 
third position then (a) the device which was in third posi- 
tion, displaying page #2, now displays page #1 , and (b) 
the color display, which was displaying page #1 , now 

10 displays page #2 in color. 

WRAPPING/EMBEDDING 

[0113] Definition: Manipulating one or more subre- 
15 gions of a device such that these subregions spatially 
occlude or are spatially occluded by some portion of the 
second device. 

[0114] Example: As illustrated in Figure 34, consider 
a first device 400 which contains the infrastructure to 

20 support the filtering of email. Consider a second set of 
devices 401 and 402 which implement particular email 
filters. When the user physically embeds device 401 (or 
402) into the first device 400, by which the first device 
400 now wraps the second device 401 (or 402), the par- 

25 ticular email filter supported by the second device is. 
activated. 

[01 15] As those skilled in the art will appreciate, com- 
binations of any of the foregoing described morphemes 
based on physical manipulation, spatial position, envi- 

30 ronmental conditions, or multiple interacting devices 
can be extended by participation in a morphemic "sen- 
tence". A sentence is defined as a sequence of one or 
more temporally disjoint morphemes. Typically, between 
about 1/1 0th of a second and 2-3 seconds suffices to 

35 distinguish morphemes within a sentence. Of course, in 
some circumstances and indefinite time period may 
elapse. The sentence level allows definition of a physi- 
cal manipulatory grammar by appropriate choice of a 
morpheme sequence, and corollary rules governing, for 

40 example, use of active (verb tike) morphemes, naming 
(noun) morphemes, or connectors. Just as the position 
and relation of words in a sentence define the sen- 
tence's meaning (e.g. "horse chestnut" is not the same 
as "chestnut horse"), similarly the position and relation 

45 of morphemes in a manipulatory sentence define the 
sentence's meaning. For example, in a communication 
mode a FLICK followed by a WHACK could mean 
"transfer data and erase the local copy", while a 
WHACK followed by a FLICK means "power on the 

so device and transfer data". In other settings, a FLICK or 
a WHACK could mean something entirely different. To 
better understand construction of morphemic sen- 
tences, the following examples are discussed: 

55 DATA TRANSFER SENTENCE 

[01 1 6] Consider a device which can transmit some or 
all of its information to another device. Furthermore, this 



12 



23 



EP0 899 651 A2 



24 



transmission can be done either unencrypted, or 
encrypted (to increase security). Furthermore, this 
transmission of a text+graphics document can either 
include the graphics, or omit them (to save time). Sup- 
pose that the user wishes to perform the command 
Transmit the information in document A, encrypted, 
omitting graphics, to machine B". Then a gestural 
sequence (morphemic sentence) to support this could 
be: 

DEPRESS - the user presses on a displayed repre- 
sentation of A, indicating that A is to be selected for 
an upcoming operation 

FLICK- the user flicks the device in the direction of 
device B, indicating that the operation is a transmis- 
sion to B 

FOLD - the user folds the top quarter of the device 
over the lower three-quarters, indicating that the 
transmission is to be encrypted. 
TWIST - the user twists the device about its central 
axis, indicating that the data is to be "'wrung out", 
i.e. the graphics are to be omitted 
SQUEEZE - the user squeezes the device, indicat- 
ing that they are confirming that the operation is to 
proceed. 

Note that none of these gestures, in isolation, performs 
an act - the "ensemble" of temporally separated mor- 
phemes must be interpreted in order to form the com- 
plete action. 

DRAWING MODIFICATION SENTENCE 

[0117] Consider a device which displays geometric 
shapes for user manipulation. Furthermore, one of the 
manipulations supported is to resize (or rescaie) a 
shape. Furthermore, suppose that this resizing can be 
done either aliased (jaggy) or anti-aliased (edges are 
smoothed). Suppose that the user wishes to perform 
the command "Resize shape A by 120%, about the X 
axis only, using anti-aliasing". Then a morphemic sen- 
tence to support this could be: 

DEPRESS- the user presses on a displayed repre- 
sentation of A, indicating that A is to be selected for 
an upcoming operation 

STRETCH - the user stretches some portion of the 
device, indicating that the operation is to be a 
resize. When the user starts stretching, a portion of 
the status display displays "100". The user contin- 
ues to stretch until the status display reads "120" 
RELIEF-MAP - the user "scores" the display by 
making a horizontal line of depression, indicating 
that the operation is to take place only about the 
horizontal (X) axis. 

DEPRESS - a circular thumb stroke is made in a 
different area of the device, indicating that anti- 
aliasing (smoothing the edges) is to be performed. 



DATABASE PRESENTATION SENTENCE 

[0118] Consider a device which contains various per- 
sonal information databases, such as a list of phone 
5 numbers, a list of addresses, and a calendar. Suppose 
that the user wishes the most appropriate of those data- 
bases displayed. Then a gestural sequence to support 
this could be: 

10 SPATIAL LOCATION - the user carries the device 
such that it is spatially proximate to either the tele- 
phone, the address book, or the refrigerator (where 
the family calendar is displayed), whichever is 
appropriate. 

15 DEPRESS - the user touches the device to activate 
the operation. 

The device now displays the personal information 
appropriate to that location. 

20 

DATABASE RETRIEVAL SENTENCE 

[01 1 9] To extend the foregoing example of a database 
presentation sentence, consider two computers which 
25 contain calendar databases. If the user wishes to syn- 
chronize the calendars, a suitable gestural sequence to 
support this could be: 

SQUEEZE - the user squeezes the device to acti- 
30 vates its gestural recognition capabilities 

ORBIT - the user orbits the device about the sur- 
face of the other device 3 times, indicating a desire 
to only match data for the next 3 weeks. 
MATCH - the user matches the edge of the device 
35 to the edge of the calendar, indicating a desire to 
"match" contents between the two devices. 

PRINTER/COPIER CONTROL SENTENCE 

40 [0120] Consider a device which can produce paper 
copies of documents. Suppose that the user wishes to 
tell such a device to produce a stapled, two sided copy 
of document A, enlarged to the next greater size. Then 
a gestural sequence to support this couid be: 

45 

DEPRESS - the user presses on a displayed repre- 
sentation of A, indicating that A is to be selected for 
an upcoming operation 

RIP - the user introduces a spatial discontinuity into 
so a portion of the device, indicating that the upcoming 
operation is to be a copy ("carrying away" some of 
the data). 

PINCH - the user pinches the upper left corner of 
the device, indicating that the copies are to be sta- 
55 pled. 

SQUEEZE - the user presses on the front and back 
of the device, indicating that the copy is to be two- 
sided. 
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STRETCH • the user stretches the device, indicat- 
ing that the copy is to be an enlargement to the next 
greater size. 

FACE • the device typically has its paper emitter on 
the bottom, preventing users from making copies 5 
accidentally. By facing the device such that the 
emitter is on the side, the copying operation is initi- 
ated. 

LIGHT BASED CONTROL SENTENCE »o 

[0121] Consider a device which can display docu- 
ments. Suppose the user is using the document while 
seated on a train, and wishes the document to display 
itself with a backlight when the train enters a tunnel, and is 
wishes the document to display itself in a larger font 
when the train rumbles over rough tracks. Then a ges- 
tural sequence to support this could be 

SQUEEZE - the user squeezes on the device, indi- 20 
eating that a loss of light is to be compensated for 
by a backlight. 

LIGHT - as the train enters a tunnel, the LIGHT ges- 
ture is made, and the device turns on the backlight. 
WHACK - the user briskly raps the device against 25 
the palm of their hand, indicating that their desired 
preference for adjusting hard-to-read documents is 
to increase the font size. 

VIBRATE - as the train goes over a bridge, the 
vibratory gesture is sensed. Because of the posi- 30 
tion of this VIBRATE morpheme (after the preced- 
ing WHACK gesture) in this morphemic sentence, 
the device now increases the font size on the dis- 
played text. 

LIGHT - the user puts the device into his suitcase, 35 
making the LIGHT gesture. In this context (with no 
SQUEEZE before it), the LIGHT gesture causes the 
device to power-off its display. 

[0122] To better appreciate utility and construction of 40 
devices in accordance with the present invention, sev- 
eral examples of devices are now described: 

PORTABLE COMPUTER WITH SQUEEZE AND TILT 
CONTROL 45 

[0123] A handheld portable computer 500 (e.g. a 
3Com ® PaimPilot ® ) capable of being fitted with 
deformable, pressure sensitive edging 504 is schemati- 
cally illustrated in Figures 35 and 36. The computer 500 so 
supports a name and address software application, pro- 
viding a user viewable name-address entry field on dis- 
play 503. In this embodiment, a user can squeeze the 
deformable, pressure sensitive edging 504 (squeeze 
arrows 507) of the computer 500. In response, the ss 
name and address software application causes the dis- 
play 503 to animate by slowly incrementing (scrolling) 
through the name list from "A" towards .T entries. 



When the user squeezes edging 504 again, the soft; 
ware application stops the scrolling animation. Scrolling 
functionality is further enhanced by the use of a tilt sen- 
sor, which allows the computer's behavior to mimic con- 
ventional rotatable address books. If the computer 500 
is tilted away from the 45 degree angle at which some- 
one might typically hold it, the scrolling rate is 
increased. In this application, the closer the computer 
500 was tilted towards the user (as indicated by arrow 
506 of Fig, 36), the faster the scroll rate toward *Z\ 
However, if a user tilted the computer 500 back past the 
neutral 45 degree position (as indicated by arrow 506 in 
Figure 36), the animation would move backwards with a 
velocity related to the magnitude of tilt In this way it was 
possible for a user to search for items in a long list in a 
very natural way, while only using one hand. 
[01 24] In an alternative mode schematically illustrated 
in Figure 37, scrolling speed can be completely control- 
led by pressure. The greater the squeeze pressure 
(arrows 537), the faster the list scrolls. Release of the 
applied pressure causes the scrolling to halt. In this 
alternative user interface strategy, applicatipn tilt (as 
indicated by orthogonal tilt arrows 530 and 532) could 
be used to change the direction of the scrolling through 
the list, allowing a user to search portions of a large two, 
dimensional data set (schematically illustrated as a data 
plane 520) that is not entirely visible either horizontally 
or vertically on display 503. By simply tilting the display 
503 of computer 500 as if it were a window through 
which the data plane 520 can be viewed, any particular 
portion of the data plane (for example, data subset 524) 
can be viewed. As will appreciated, in both the foregoing 
modes the speed of scrolling, the specific neutral tilt 
angle, and required pressures to initiate scrolling 
changes can be adjusted to fit a particular user. 

PORTABLE COMPUTER WITH HANDEDNESS 
DETECTION 

[0125] Pressure sensors have been added to aug- 
ment a conventional keyboard 551 enabled user inter- 
face to a hand holdable Windows ® CE class computer 
550 (i.e. a Cassio ® Cassiopia ®) schematically illus- 
trated in Figures 38 and 39. In this embodiment, user 
handedness was determined by using pressure sensors 
positioned on a right back-edge and a left back-edge of 
computer 550. User studies have found that the differ- 
ence in pressure between the right and left side gave a 
direct indication of handedness of a user. As respec- 
" tively illustrated in Figures 33 and 39, handedness was 
used to justify formatted text 554 to the left (Fig. 38) or 
right (Fig. 39). thereby allowing more space 555 on dis- 
play 553 for an electronic annotation pen to be used to 
mark-up the text 

[01 26] For both the embodiment of the invention illus- 
trated by Figures 35-37, and the foregoing embodiment 
illustrated in Figures 38 and 39, material deformation of 
a spongy, elastic, or otherwise deformable material 
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must be measured. Although various techniques can be 
used to measure material deformation, including those 
based on imaging or fluid volumetric changes, one par- 
ticularly useful technique is based on the use of pres- 
sure transducers. Commercially available sensors 5 
measure pressure (indicative of material deformation) 
by converting a pressure change to a change in electri- 
cal characteristics. For example, inexpensive sensors 
that change resistance in response to pressure can be 
obtained in a variety of shapes and sizes, including w 
paper thin sensors, and easily bendable sensor strips. 
Sensors of this kind can be customized to any particular 
shape or form that a gestural Ul might require. The 
change in resistance is usually linearly related to pres- 
sure, with the sensor generally being placed in a poten- 75 
tial divider network to model the pressure as a change 
in potential. For a practical circuit the resulting signal 
needs to be amplified, buffered and translated such that 
the change in value from minimum pressure to maxi- 
mum pressure spans a useful range. The modified sig- 20 
nal can now be fed into an analog to digital converter 
(ADC) to produce a digital representation of pressure. 
An 8-bit ADC can typically be used for most applica- 
tions, however, if greater sensitivity to pressure changes 
is required, a higher resolution ADC (e.g 16-bit ADC) 25 
can be used. As will be appreciated, the ADC could be 
memory mapped into the processor's address space as 
a peripheral, or alternatively supplied as a retrofitted 
pressure interface to existing computers capable of 
benefiting from this system. Since an RS232 connection 30 
is an interface that is almost universally available on 
portable computers, one strategy is to have the parallel 
output of the ADC converted into a serial RS232 frame 
using a seriaiizer such as a UART and then level-shift 
and buffer the signal as specified by the RS232 stand- 35 
ard. At the computer end of the serial interface, another 
level-shifter and UART, the output of which is readable 
by the processor, performs the serial-to-parallel conver- 
sion. 

[0127] As described with reference to Figure 40, in 40 
realizing a working system a microcontroller 564 (a Sig- 
netics 87c752) with an ADC built in to combine many of 
the I/O tasks in a single chip can be used in combination 
with level shifter 566 (a MAX3223). This approach has 
the advantage that intelligent processing of the input 45 
signal is possible with software. Tilt measurement was 
provided by a tilt sensor 567 connected to a buffer/amp 
562 to supply an analog signal to microcontroller 564. 
The pressure measurement can also be encoded within 
a protocol across the serial link. This particular micro- so 
controller 564 has five ADC inputs, but by making use of 
eight digital control lines, it is possible to use only one 
ADC input and one buffer amplifier 561 , to measure up 
to eight pressure points with pressure sensors 565. This 
is achieved by using the control lines to select only one 55 
sensor at a time and take a reading for each using a sin- 
gle input to the ADC. After eight sensors have been 
selected, eight readings are acquired in memory. The 



design is practical because the microcontroller can take 
measurements and make analog to digital conversions 
at a rate far higher than is needed to communicate with 
a host computer 569. 

[0128] For scrolling or handedness based software 
applications. 16 levels were determined to be adequate 
for representing a pressure measurement. In order to 
have a high data throughput to the host computer 569. 
each measurement was encoded in one byte of an 
RS232 frame such that the four lowest bits were the 
pressure representation and the highest four bits were 
the sensor ID. Each frame of the RS232 data was there- 
fore completely self contained. Of course any protocol 
that limits itself to a specific number of devices in its 
address space will eventually have the problem that 
someday applications might be designed that need to 
reference far more devices than are supported. The 
solution used in this protocol is to reserve the sensor ID 
number 15 as a special value that can extend the 
semantics of the encoding to include an arbitrary 
number of bytes representing a sensor or value. For the 
described software applications, the commonly used 
RS232 frame format (1 start, 8 data no parity, 1 stop bit 
at a baud rate of 9600) was chosen. 
[0129] In operation, the host computer 569 deter- 
mined handedness by utilizing information about the 
current pressure exerted on two subregions - one, on 
the back of the device, occupying roughly the left half, 
and the other, also on the back of the device, occupying 
the right half. The current pressure values were con- 
verted from analog to digital form, such that a digital 
value of 0 (zero) represents no pressure, and a digital 
value of, for example, 15 represents maximum pres- 
sure. The detection circuitry then proceeds: 

if (left_sensor is high AND right_sensor is high) 
then 

conclude that the user is gripping the device 
with both hands 

else if (left_sensor is high AND right_sensor is low) 
then 

conclude that the user is gripping the device 
with the left hand 

only 

else if (left_sensor is low AND right_sensor is high) 
then 

conclude the user is gripping the device with 
the right hand only 

else if (left.sensor is low AND right_sensor is low) 
then 
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conclude that the user is gripping the device 
with neither hand 

Also, to optimize communication, pressure values are 
only sent when they change. In order to account for jitter 5 
and error in the pressure sensors, a sensor is only con- 
sidered 'high' if its value is higher than some minimum 
threshold (e.g. 'Z on the 0 to 15 range). 

SCANNER/PRINTER/COPIER SUPPORTING EDGE io 
DEFORMABLE DISPLAY 

[0130] A scanner/printer/copier device 570 is sche- 
matically illustrated in Figure 41. As seen in Figure 41 
(and in more detail in Figure 42) the device 570 sup- is 
ports a display 574 having a deformable edge 572, with 
the combination roughly shaped like a piece of paper, in 
operation, a user can place a written document in the 
device 570 for scanning. An electronic version of the 
scanned document is displayed (i.e. as text 575 in Fig- 20 
ure 42) on the display 574. By outwardly pulling the 
deformable edge 572 as indicated by arrow 577, a user 
can instruct the device 570 to resize the document 
before printing or copying. Pinching opposing sides of 
the deformable edge 572 (arrows 578) can further 25 
instruct the device 570 to print double sided copies. As 
will be appreciated, various other morphemes can be 
used to provide a simple interface for interacting with 
device 570. 

30 

TILEABLE AND STACKABLE PORTABLE DISPLAYS 

[0131] Multiple autonomous display tiles having an 
onboard display controller and at least one surface con- 
sisting substantially of a conventional display are partic- 3$ 
ularly useful for the practice of various aspects of the 
present invention. Such tiles can be interconnected in 
response to various morphemes such as TOUCH, 
FLICK, RELATIVELY ALIGN, or WHACK, or even can 
be operated without substantial user mediated morphe- 40 
mic input in certain situations. 
[01 32] Advantageously, tile positioning can be used as 
an interface specifier in and of itself. For example, each 
display tile can contain a video segment in independent 
memory. Shuffling or reorganizing the tiles can allow 45 
users to physically manipulate the sequence of video 
segments to affect a physically manipulatable video 
editing system. Using the card analogy, tiles can be 
used to re-sequence documents, pages within a docu- 
ment, audio annotations, voice mail, or other temporal so 
media contained within the tiles. The resultant 
sequence can then be played as a whole by using the 
tiled structure as a unit. 

[01 33] For purposes of the present invention, display 
tile array configurations can be categorized as follows: ss 
[01 34] Close-packed display tile array 600 (Figure 43) 
in which tiles 602 are arranged to span a surface 610 in 
closely abutting, but not overlapping relationship, such 



that the continuous display area is maximized (i.e. 
seamlessly). As will be appreciated, the tiles could 
themselves form a freestanding continuous surface, or 
they could be disposed on a table or other suitable sup- 
port. Each of the tiles 602 supports a display 604 sized 
to substantially cover a front surface of each tile 602. In 
certain embodiments, a back surface of each tiie 602 
can also support a display. Advantageously, this would 
allow for creation of free standing displays with images 
visible on the front and back. The surface 610 can be a 
plane, a sphere, or any arbitrary shape that permits til- 
ing. 

[01 35] Loose-packed display tile array 620 (each tile 
622 having a display 624 as seen in Figure 44) in which 
tiles 622 are slotted into a lattice pattern (indicated by 
dotted lines 625). Each tile can be considered to be sit- 
uated in a regular bounded lattice slot (namely, the lat- 
tice slot that contains a defined center of each of the 
tiles) within the lattice where the dimensions of the 
bounded slot are no more than a few times the maxi- 
mum dimension of the tile, and there is no more than 
one tile in any slot. Within the lattice regions any of the 
tiles can be positioned arbitrarily and still retain the 
same inter-relationship with the group. It is still possible 
for tiles to touch one another at the boundaries of two or 
more lattice slots, however, this is not a requirement. 
[01 36] Free-format display tiles 630 (each tile 632 hav- 
ing a display 634 as seen in Figure 45) are similar to 
loose-packed display tiles 620 where the size and 
shape of the lattice slots may vary freely (e.g. the 
dimension of a lattice slot may be many times the maxi- 
mum dimension of any of the participating tiles). The 
only constraint on the arrangement is that there must be 
no ambiguous relationships about the relative connec- 
tivity of the each of the tiles 632. That is to say, a tile that 
is to display the next piece of information to one side of 
another tile must be unique and not be confused with 
the task of another tile in the tiling lattice. 
[01 37] 3D display tiles (packable display tiles) are cre- 
ated by extending the three foregoing display tile cate- 
gories. However for close packed tiling of display tiles, 
packed devices in the center of a 3D structure would not 
be available as the user interface. This may not matter 
as the surface of the 3D shape will expose a area that 
will have unique affordances for some classes of appli- 
cation. For instance, cubic tiles packed into the shape of 
a large cube can display, using the six faces of the larger 
cube, the various projections that could be rendered by 
viewing a 3D CAD drawing from each degree of free- 
dom. 

[0138] As will be appreciated, display tiles need not be 
recti-linear but can be hexagonal, circular, or of arbitrary 
shape and size. The size of tiles need not be constant 
for all tiles within a larger structure. Tiles need not be 
aligned but may require proximity to indicate adjoining 
points or edges. 

[01 39] Ttles need not be in physical contact to define 
they are joined in a group activity but instead this may 
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be a programmed function. The connectivity can also be 
discovered through a radio network from a coordinating 
server or from a distributed algorithm that draws in as 
many computers as necessary for a task, using the 
wireless network to negotiate the resources it needs. 
The tiled computers may also be connected by wired 
networking systems, although in cases where the topol- 
ogy might need to rapidly change, this is not as desira- 
ble as a wireless system. An example of a wired 
networking system of this type is a system that uses the 
internet to include many computers in a single task, one 
in which each computer knows its relative position even 
though they may be separated by rooms or in the 
extreme cases buildings, cities or countries. However, in 
general the most useful case of display tiling is when the 
tiles are proximate enough that they can all be viewed 
by one person to create a display medium whereby the 
viewing experience is an enhancement over using a sin- 
gle display. Accordingly, tiles can operate as either a 
single larger contiguous structure or they can retain 
individual properties and independent functions or a 
combination of both, (e.g., jumbo-tron like function, 
12x12 small displays as used in a TV studio , or picture 
in-picture features found in commercial TVs or editing 
suites). 

[0140] Depending on the type of packing, various 
schemes can be used to allow for permanent, intermit- 
tent, or even one-time communication between display 
tiles. For example, close packed tiling can make use of 
wired connectivity between computers or could use a 
variety of wireless or optical communication technolo- 
gies. In the case of wired connectivity, edge mounted 
conventional plug and socket connectors may be used 
to create a rigid tiled array. Plug and socket systems 
lend themselves to parallel connections for bulk and 
high-speed data transfer. They also provide a conven- 
ient method of power distribution, which can allow for 
one of the tiles providing a power source that supports 
the rest of the array. A plug and socket connection 
between tiled computers 692 in a tiled computer array 
690 (with both data and power transfer shown) is sche- 
matically illustrated by lines 695 in Figure 48. 
[0141] The exact design and locations of electrical 
connectors depends on the intended use. and subse- 
quently , the shape of tile components. For applications 
where large seams between displays are acceptable , 
simple rigid connectors could be attached to the center 
of each edge, providing connectivity to all the surround- 
ing tiles. Other applications might require a more com- 
plex design. For example, applications which require 
high-quality information display on a seamless array 
surface (i.e. tiled blueprints), might employ spring- 
loaded contacts on the four edge connectors. The 
spring mechanism allows all array connections to be 
made below the display surface, while tiles can still be 
inserted and removed from the interior of the array. 
Removal of tiles could be triggered by some host signal 
which actuates a release of the spring and pops the tile 



out of the array 

[01 42] Serial connectivity can also be used in a close 
packed arrangement such as that illustrated in Figure 
43. It has the advantage that fewer connections have to 

5 be made and in practice it might be more reliable. How- 
ever the net bandwidth between tiles will be less than a 
parallel system. Serial communication lends itself to 
optical and wireless systems thus removing the need for 
any physical connections. For optical technologies 

10 alignment of the transmitter and receiver is still impor- 
tant although the clever use of light pipes and lens cap- 
ture techniques can introduce more flexibility. Wireless 
systems can use many different bands of the EM spec- 
trum (kHz, MHz, GHz), utilizing a variety of modulation 

75 techniques (amplitude modulated, frequency modu- 
lated, or those based on code division multiple access) 
and operate at a range of transmitter powers. There is 
no longer a need for direct alignment if the system is 
designed with suitable communication tolerances. The 

20 transmitter range plays a crucial part in the design. If 
there is only enough power in an emitted signal to be 
picked up within a few millimeters of a tile edge, then the 
signals will be isolated, the topology will be defined by 
the physical connectivity and the complexity of design- 

25 ing the system to avoid interference from neighboring 
signal sources will be minimized. However, an alterna- 
tive design is to use more powerful radios. In this case 
all tiles will be able to contact all other tiles and inter-tile 
connectivity needs to be defined by another parameter. 

$o Signal strength can be used or, more deliberately, infor- 
mation that relates the ID of a tile to a spatial map (per- 
haps held in one master tile) describing the position of 
ail tiles in the tile array. In this system it is also neces- 
sary to minimize inter-tile interference. For digital 

35 packet-data systems that operate at the same fre- 
quency, carrier sense multiple access (CSMA-CD or 
CSMA-CA) systems are well known techniques to solve 
this problem. Other solutions involve tiles using different 
frequencies, with frequencies reused depending on the 

40 power of the transmitters. This is the technique used by 
traditional cellular telephones. Yet another approach is 
to use Code Division Multiple Access (CDMA) that 
relies on the overlaying of signals in the same region of 
the EM spectrum, a technique known as spread spec- 

45 trum modulation. 

[0143] For loose packed display tiles, the wireless 
techniques described above in connection with close 
packed tiling generally become essential to the imple- 
mentation. However, a special case of loose packing 

so exists in which each edge of a tile does make contact 
with every other surrounding tile, except that it may only 
be a single point of contact and not at a accurately 
defined place. A wired version of this system can be 
built in which the entire edge of each tile is a serial con- 

55 nection including one of the two vertices that define the 
edge. Communication in each direction can be achieved 
by a variety of commercially available techniques, 
including use of a one-wire interface (plus a ground) for 
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bidirection communication between tags and a reader. 
Note that the ground connection for a tile arrangement 
can be derived by sharing a common ground connec- 
tion through the surface the tiles are laid out on. For 
example, the surface couid be made of a metal sheet. 5 
The system may be further enhanced by ensuring that 
the edge contact is made of a magnetic material and the 
vertices have a magnet embedded in the end. Such an 
arrangement ensures that there will be a good electrical 
contact between the transmitter and the receiver. w 
[0144] The receiver can also derive power from an 
electrically transmitted signal by bridge-rectifying it and 
storing the collected charge in a capacitor for use by its 
own electronics. Thus, power distribution can also be 
included in a one-wire interface. In this way flexible con- is 
nectivity can be achieved to support the rapid and con- 
venient rearrangement of tiles in which only the 
minimum amount of care needs to be taken in setting up 
connectivity. 

[0145] Loosely packed tile displays such as illustrated 20 
in Figure 45 may require that the display surfaces use a 
best effort algorithm to present a unified display with all 
the sections of the displayed image bearing the correct 
spatial arrangement to each other even though, in the 
case of rectangular tiles, they might not be registered 25 
vertically or horizontally and have an offset angle rela- 
tive to each other. In order to implement the desired tile 
display algorithm, not only is the relative arrangement of 
tiles important but the exact offset (distance and angle) 
from each other is also important. There are several 30 
methods that are suitable for automatically determining 
offsets between loosely packed tiles. For example, as 
illustrated with respect to Figure 46 (showing communi- 
cating tiles 652 and 654, with respective displays 651 
and 653), optical encoding 660 along an edge can be 35 
used to identify tile orientation. It is possible to use a 
binary coded optical pattern along an edge which is reg- 
ular and encodes the distance from a vertex at any 
point. A tile that abuts, or is relatively aligned, can read 
this pattern using optical sensors 658 and 659 and 
determine the display offset in the direction of the edge. 
Alternatively, as illustrated by Figure 47 (showing com- 
municating tiles 672 and 674), it is possible to use radio 
based techniques that rely on signal strength triangula- 
tion. Each vertex 675 or 676 of a tile 672 can contain a 
wireless transmitter and a receiver. If these vertices 
transmit a short characteristic wireless signal at well 
known times, a nearby tile 674 can use the receivers sit- 
uated at its own vertices 680, 681, 682, and 683 to tri- 
angulate the position of each transmitting vertex relative 
to them by measuring the relative delay by which the 
signal was received. After two of the vertices of the orig- 
inal transmitting tile 672 have signaled, the adjoining tile 
674 can determine its exact position and orientation 
within their local region of a tile array. The transmitting 
and receiving tiles can now swap roles, with the result 
that both tiles will know their relative position. This proc- 
ess can be extended across the tile array. 



[0146] Free format tiling differs from loose-packed til- 
ing in that there are no proximity, or regular format, con- 
straints. To demonstrate how a free format tiling system 
might work, the following example is described. Imagine 
a number of laptop computers each fitted with a Global 
Positioning System (GPS) and a radio modem. Each 
laptop can determine its position (with acceptable error) 
and communicate it to all the other laptops by contact- 
ing them through the radio modem. After some period of 
time the laptops will all know their relative location and 
absolute location. !f any computers changes its location, 
it can update the local neighbors to ensure there is 
enough understanding in the array to consider the com- 
puters are in a known tiling configuration even though 
they are not close to each other and in fact they may be 
in different geographic regions. An application that 
might use this free format tiling system is one that 
wishes to ensure information is being correctly sent in a 
uniform and dispersed fashion across a very large area. 
For example, the information that each laptop computer 
receives could be instructions to release an amount of a 
pesticide used to control an insect that does damage to 
commercial crops. If the pesticide is released in too high 
a concentration in any locality, it may be hazardous to 
human health. The tiled approach allows the computers 
to roam (e.g. in the back of a truck), displaying informa- 
tion as to the type and concentration of pesticide that is 
to be released given their current relative proximity. 
[01 47] Various conventional algorithms can be used to 
support distribution of information between autonomous 
tiled displays. These algorithms assume a system by 
which there is a master controller generating data to be 
displayed. There is also a large array of tiled computers 
that the system will use to display visual data and/or 
process information. Each computer in the tiled array 
contains a unique ID. It is the job of the master to split 
the data into pieces that each computer can display and 
for this information to be packaged along with the ID of 
the target computer. The algorithms below describe 
how the information travels from the master to a desti- 
nation display tile: 

Daisy chain routing 

[01 48] Display tiles are arranged to have a logical con- 
nectivity with each other so that each one only transmits 
information to the next in a predefined line. The comput- 
ers are said to be daisy chained to each. Any informa- 
tion sent to the start of the chain contains an ID and the 
first computer in the chain compares it to its own ID. If it 
matches, it acts on the data. If not, it sends the data on 
to the next computer in the chain until it finds its destina- 
tion. 



[0149] In N-ary routing the path to the destination is 
contained in the ID of the device. A simple routing 
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example is schematically illustrated with refrence to the 
direction arrows 695 in Figure 48, which shows quater- 
nary routing in physically connected tileable displays. In 
quaternary routing, an array is conceptually arranged as 
a quaternary tree with each node having an input and 
three outputs. In this system each pair of bits of the ID 
contains a routing command. A 0 indicates send the 
packet to the first output, a 1 indicates the second out- 
put, a 2 indicates the third output, and 3 indicates no fur- 
ther transmission. There is also a count that gets 
decremented by each node to tell successive nodes the 
bit number that is currently being considered and when 
the packet has reached its destination. In this way pack- 
ets are forwarded from node to node with a simple 
choice at each stage until they reach their target display. 
As will be appreciated, it is possible to design N-ary sys- 
tems with more than three outputs (a power of two is 
usually convenient for implementation (e.g. 4, 8, 16 ...)). 

Flooding 

[0150] Flooding has no predefined routing structure. 
The computer that takes the first packet from the master 
checks to see if it has the correct ID. If not, the packet is 
sent out on all links to which it has not yet sent or 
received that packet. The result is a flood of copies of 
the packet across the array eventually reaching its des- 
tination. The packets must also have a maximum hop 
count to ensure they eventually are removed from the 
system. The disadvantage of this approach is that many 
more tiles are burdened with the processing of 
unwanted data than in the previous two schemes, which 
may impact the overall efficiency of the system. 

Hot Potato 

[0151] The Hot Potato algorithm is similar to the 
Flooding algorithm, except a retransmitted packet is 
sent out on only one output that is either chosen ran- 
domly or is the least busy. The process stops when the 
packet reaches the correct tile. The time that a packet 
takes to reach its destination is not deterministic. 

Claims 

1. A plurality of tiieable devices for transferring data, 
comprising 

a first device having a display, a processor, and 
a first communication module for transferring 
data, 

a second device having a display, a processor 
and a second communication module for trans- 
ferring data, 

a third device having a display, a processor and 
a third communication module for transferring 
data, 

wherein the first device is connected in sub- 



stantially simultaneous communication with the 
second device and the third device to pass data 
based on spatial positions of the respective first 
device, second device, and 

5 

third device. 

2. The plurality of tileable devices of claim 1 , wherein 
the first device maintains information about spatial 
w position of the second device and the third device, 
and transfers differing data to the second device 
and the third device based on that maintained spa- 
tial position information. 

is 3. The plurality of tileable devices of claim 1 , wherein 
the first, second and third communication modules 
respectively support optical links. 

4. The plurality of tileable devices of claim 1 , wherein 
20 the first, second and third communication modules 

respectively support radio links. 

5. The plurality of tiieable devices of claim 1 , wherein 
the first, second and third communication modules 

25 respectively support physical electrical connec- 
tions. 

6. The plurality of tileable devices of claim 1, each 
first, second and third device respectively further 

30 comprising a first, second, and third deformable 
piece, with intercommunication between at least 
two of the devices being triggered by a touching 
action between devices. 

35 7. The plurality of tileable devices of claim 1. each 
first, second and third device respectively further 
comprising a first, second, and third deformable 
piece, with intercommunication between at least 
two of the devices being triggered by a stacking 
40 action between devices. 

8. The plurality of tileable devices of claim 1, each 
first, second and third device respectively further 
comprising a first, second, and third deformable 

45 piece, with intercommunication between at least 
two of the devices being triggered by a tiling action 
between devices. 

9. A dynamic array of devices, the array comprising, 

50 

a plurality of devices, each device having a dis- 
play maintainable in a user determined rela- 
tionship to each other, and each device having 
a determinable information state, 
55 a plurality of position detectors, each device 

having at least one of the plurality of position 
detectors to determine its position relative to 
one or more of the plurality of devices, 
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wherein determinable information state of each 
of the plurality of devices is modified upon its 
change of position relative to one or more 
devices. 

5 

10. A tileable display system, comprising 

a first device having a display, a processor, and 
a first communication module for transferring 
data. 10 
a second device having a display, a processor 
and a second communication module for trans- 
ferring data, 

wherein the first device is connected in sub- 
stantially simultaneous communication with the is 
second device to pass data based on spatial 
positions of the respective first device and sec- 
ond device. 
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Fig. 6 
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Fig. 8 
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Fig. 15 



250 




Fig. 16 
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Fig. 23 



33 



EP0 899 651 A2 




34 



EP 0 899 651 A2 




Fig. 26 
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Fig. 30 
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Fig. 31 




Fig. 32 
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Fig. 36 
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Fig. 37 
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Fig. 39 
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Fig. 42 
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